Method and device for enabling virtual reality interaction with gesture control

ABSTRACT

The present invention is a method for enabling virtual reality interaction with gesture control, comprising the following steps: displaying a photographed second image in response to a gesture detecting signal; recognizing an actual gesture action in the second image and converting the same into a graphic pointer; and displaying a first image and displaying the graphic pointer at a position corresponding to the first image for interaction. A user may conduct control directly with a gesture to enable virtual reality, thereby solving the problem where the user cannot see an actual scenario as well as increasing the ease of use for virtual reality.

CROSS-REFERENCE TO RELATED APPLICATIONS

This Non-provisional application claims priority under 35 U.S.C. §119(a) on Patent Application No(s). 201710004798.0 filed in People'sRepublic of China on Jan. 4, 2017, the entire contents of which arehereby incorporated by reference.

BACKGROUND OF THE INVENTION Field of the Invention

The present invention relates to the technical field of virtual reality,and in particular to a method and device for enabling virtual realityinteraction with gesture control.

Description of the Related Art

In 2016, the virtual reality industry (i.e. the so-called VR industry)remained a focus around the world, and virtual reality devicees, i.e. VRdevices, from all the leading manufacturers came into the market oneafter another. Most of the VR devices for information interactionincorporate a control handle or a touch pad on the whole and thus arehigh in price, common users generally can only accept low-price VRdevices due to a price limit, and the low-price VR devices may be onlyused for displaying an image on a mobile terminal and cannot enableinformation interaction for the users.

In this case, Hand Gesture for Mobile VR, i.e. a gesture for a virtualreality device turns up as the times require. However, when answering acall or checking a short message while watching a VR image, a usercannot acknowledge the surrounding environment, move a hand into aregion detectable for a camera, and execute an action with the handaccurately due to the wearing of a helmet, in case of not adding anyhardware to the current Hand Gesture for Mobile VR.

BRIEF SUMMARY OF THE INVENTION

To solve the technical problem that a user cannot acknowledge thesurrounding environment and accurately move the hand to a regiondetectable for a camera due to the wearing of a helmet duringinformation interaction, the present invention provides a method forenabling virtual reality interaction with gesture control.

A method for enabling virtual reality interaction with gesture controlcomprises the following steps:

displaying a photographed second image in response to a gesturedetecting signal;

recognizing an actual gesture action in the second image and convertingthe actual gesture action into a graphic pointer; and

displaying a first image and displaying the graphic pointer at aposition corresponding to the first image for interaction.

As an implementable means, recognizing the actual gesture action in thesecond image and converting the actual gesture action into the graphicpointer may comprise the following steps:

capturing moving block masses appearing in the second image, analyzingthe moving block masses to obtain several block masses approximate to ahand, and selecting all the block masses approximate to the hand ascandidate hand regions;

determining a hand shape in each candidate hand region, and if aplurality of the candidate hand regions pass verification, selecting asum of the candidate hand regions passing the verification as aparticular region; and

converting the actual gesture action in the particular region into agraphic pointer.

As an implementable means, the first image may be presented before agesture detecting signal is detected; and

the displayed photographed second image may contain the first image.

As an implementable means, that the displayed photographed second imagecontains the first image may refer that the first image and the secondimage are superimposed with an Alpha Blending algorithm.

As an implementable means, the second image may be preprocessed as thesecond image is photographed.

As an implementable means, the second image may be preprocessed afterthe second image is photographed.

As an implementable means, the preprocessing performed on the secondimage may comprise at least one of zooming, white balance adjusting,exposure adjusting and image de-noising.

An device for enabling virtual reality interaction with gesture controlcomprises a second image displaying module, a pointer recognizing moduleand an interaction displaying module;

the second image displaying module is used for displaying a photographedsecond image in response to a gesture detecting signal;

the pointer recognizing module is used for recognizing an actual gestureaction in the second image and converting the actual gesture action intoa graphic pointer; and

the interaction displaying module is used for displaying a first imageand displaying the graphic pointer at a position corresponding to thefirst image for interaction.

As an implementable means, the pointer recognizing module may comprise acapturing and analyzing module, a hand shape detecting module and agraphic pointer converting module;

the capturing and analyzing module is used for capturing moving blockmasses appearing in the second image, analyzing the moving block massesto obtain a plurality of block masses approximate to a hand andselecting all the block masses approximate to the hand as candidate handregions;

the hand shape detecting module is used for determining a hand shape ineach candidate hand region, and if the plurality of candidate handregions pass verification, selecting a sum of the candidate hand regionspassing the verification as a particular region; and

the graphic pointer converting module is used for converting the actualgesture action in the particular region into the graphic pointer.

As an implementable means, the interaction displaying module may beconfigured so that the first image is presented before the gesturedetecting signal is detected.

As an implementable means, the second image photographed throughinteraction display may contain the first image.

As an implementable means, the interaction displaying module may beconfigured so that the displayed photographed second image contains thefirst image refers that the first image and the second image aresuperimposed with an Alpha Blending algorithm.

As an implementable means, the interaction displaying module may beconfigured so that the second image is preprocessed as the second imageis photographed.

As an implementable means, the interaction displaying module may beconfigured so that the preprocessing performed on the second imagecomprises at least one of zooming, white balance adjusting, exposureadjusting and image de-noising.

Compared with the prior art, the present invention has the followingadvantageous effects:

the method for enabling virtual reality interaction with gesture controlof the present invention comprises the following steps: displaying aphotographed second image in response to a gesture detecting signal;recognizing an actual gesture action in the second image and convertingthe same into a graphic pointer; and displaying a first image anddisplaying the graphic pointer at a position corresponding to the firstimage for interaction. A user can conduct control directly with agesture to enable virtual reality, thereby solving the problem where theuser cannot see an actual scenario as well as increasing the ease of usefor the virtual reality.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic diagram of a flow process of a method for enablingvirtual reality interaction with gesture control of an embodiment of thepresent invention;

FIG. 2 is a schematic diagram of a flow process of recognizing an actualgesture action in a second image and converting the same into a graphicpointer of the present invention;

FIG. 3 to FIG. 6 are schematic diagrams of specific applicationscenarios of embodiments of the present invention;

FIG. 7 is a schematic diagram of an device for enabling virtual realityinteraction with gesture control of the present invention; and

FIG. 8 is a schematic diagram of a pointer recognizing module of thepresent invention.

DETAILED DESCRIPTION OF THE INVENTION

The above and additional technical features and advantages of thepresent invention will be described clearly and completely below incombination with accompanying drawings, and it is obvious that theembodiments described below are merely part of rather than all of theembodiments of the present invention.

Regarding an overview of application scenarios, the present invention isenabled depending on a virtual reality device; when a user uses thevirtual reality device (the VR device), there is a gesture detectingsignal accessed suddenly; at the moment, the second image photographedis displayed, and an actual gesture in the second image is convertedinto a graphic pointer; and at the moment, a first image is displayed,moreover, the graphic pointer is displayed at a position correspondingto the first image, and the virtual reality interaction is controlledwith a gesture action.

Optionally, the first image appears when the second image is displayedor after the second image disappears.

That is, the steps to be described next in the methods of theembodiments of the present invention may completely depend on thevirtual reality device or an device similar to the virtual realitydevice for execution.

An exemplary method is as follows:

as is shown with reference to FIG. 1, a method for enabling virtualreality interaction with gesture control comprises the following steps:

Step S1, displaying a photographed second image in response to a gesturedetecting signal;

Step S2, recognizing an actual gesture action in the second image andconverting the same into a graphic pointer; and

Step S3, displaying a first image and displaying the graphic pointer ata position corresponding to the first image for interaction.

Further, recognizing an actual gesture action in the second image andconverting the same into a graphic pointer comprises a specific processas follows:

S201, capturing moving block masses appearing in the second image,analyzing the moving block masses to obtain several block massesapproximate to a hand, and selecting all the block masses approximate tothe hand as candidate hand regions;

S202, determining a hand shape in each candidate hand region, and if aplurality of the candidate hand regions pass verification, selecting asum of the candidate hand regions passing the verification as aparticular region; and

S203, converting the actual gesture action in the particular region intoa graphic pointer.

With the process for converting the actual gesture action into thegraphic pointer using the method for enabling virtual realityinteraction with gesture control and the actual gesture actionspecifically recognized in the second image, a user may conduct controldirectly with a gesture to enable virtual reality, thereby solving theproblem where the user cannot see an actual scenario as well asincreasing the ease of use for the virtual reality.

Optionally, the first image may be displayed before the gesturedetecting signal is detected, the photographed second image is displayedwhen the gesture detecting signal is detected, and at the moment, thephotographed second image that is displayed contains the first image.

Embodiment 1

A method for enabling virtual reality interaction with gesture controlis illustrated in detail with reference to FIG. 3 to FIG. 6 as schematicdiagrams of scenarios.

In the present embodiment, as shown in FIG. 3, a user watches the firstimage; as shown in FIG. 4, the user has a call accessed, that is, once agesture signal is detected, the photographed second image starts to bedisplayed, however, the second image displayed at the moment issubjected to processing and contains the first image therein; as shownin FIG. 5, an actual gesture action appears in the second image and isconverted into a graphic pointer; and as shown in FIG. 6, the secondimage disappears, the first image is displayed, the graphic pointerappears at a position corresponding to the first image to track theactual gesture action, and the user controls the position of a virtualpointer by moving the hand so as to control the action execution of thepointer with a gesture action.

In FIG. 5, the actual gesture action appears, in FIG. 6, the graphicpointer appears, and with reference to FIG. 2, a specific method forconverting the actual gesture action into the graphic pointer comprisesthe following steps:

S201, capturing moving block masses appearing in the second image,analyzing the moving block masses to obtain several block massesapproximate to a hand, and selecting all the block masses approximate tothe hand as candidate hand regions;

S202, determining a hand shape in each candidate hand region, and if aplurality of the candidate hand regions pass verification, selecting asum of the candidate hand regions passing the verification as aparticular region; and

S203, converting the actual gesture action in the particular region intoa graphic pointer.

Step S201 has a specific process as follows: some moving block massesare obtained with skin color and motion detection methods, some movingblocks that cannot be hands are filtered according to information suchas color and shape of the moving block masses; the moving block massesremained after filtering that are possibly hands are subjected tomovement predicting and tracking and are analyzed in terms of movingspeed, trace and other information thereof to obtain some moving blockmasses that are most possibly hands, and regions in which these movingblock masses appear are taken as candidate regions to be detected in anext step;

Step S202 has a specific process as follows: in each candidate region, aplurality positions and a plurality of dimensions are subjected to theextraction of texture features and gradient features, the extractedtexture features are sent into a classifier well trained off-line forjudging a hand shape, and if a plurality of positions pass verification,non-maximal suppression needs to be performed on these candidate regionsto finally output a rectangular frame containing the actual gestureaction; and

the actual gesture action is finally converted into a graphic pointer inthe rectangular frame.

After the actual gesture action is converted into the pointer, the handneeds to be tracked and a hand action needs to be recognized to furthercontrol the position of the virtual pointer by moving the hand and tocontrol the action execution of the pointer with the gesture action.

The tracking of the hand includes movement prediction, tracking, onlinelearning and online verification, where the movement prediction is topredict the position of a current frame target according to a previousmovement status of the target; a tracking algorithm may be one such asparticle filtering, optical flow tracking and compression tracking; anonline learning module extracts the feature of a tracking target in realtime for learning, and updates the online classifier in real time; andthe online verification is to verify the current tracking target withthe online classifier, and the target is abandoned if the target is nota hand, thereby excluding possible tracking errors.

Before the hand is tracked, a predefined dynamic gesture is configured,the gesture action is recognized while the movement and change of thehand are detected, and compared with the predefined dynamic gestureaction, the pointer is controlled if the gesture action is the same asthe predefined dynamic gesture.

A particular action of the hand is also sensed while the hand istracked, for example, the gesture action also needs to be made forrecognition, comparison with the predefined dynamic gesture andrecognition of the hand shape of the user, therefore, the user mayfinish some clicking actions with particular actions; while the hand istracked, the gesture also needs to be recognized for recognizing agesture type, and to determine whether the gesture changes and whetherthe gesture changes from one static gesture to another static gesture,for example from a palm into a fist, these gestures need to berecognized with a recognition method as follows:

multi-classification recognition performed with a depth neutral networkwell-trained off-line.

In this embodiment, the second image needs to be preprocessed duringphotographing, the preprocessing performed on the second image compriseszooming, white balance adjusting, exposure adjusting, de-noising and thelike; and after the preprocessing, the second image and the first imageare subjected to Alpha Blending (Alpha Blending for Transparency), andthe second image is displayed after the first and second images areprocessed, with the first image contained in the displayed second image.

If the second image is the one photographed with a depth camera, thesecond image and the first image need to be subjected to alignmentprocessing and then Alpha Blending (Alpha Blending for Transparency),and the second image is displayed after the first and second images areprocessed, with the first image contained in the displayed second image.

In one embodiment, when the user watches the first image in the VRdevice, a gesture detecting signal can be taken as being triggered ifthe user wants to play with short message or with a phone, and themethod for enabling the virtual reality interaction with gesture controlcan be used in these situations for interaction.

Exemplary Device

After the method of the exemplary embodiment of the present invention isintroduced, an device for enabling virtual reality interaction withgesture control of an exemplary embodiment of the present invention isintroduced next with reference to the drawings.

The implementation of the device for enabling the virtual realityinteraction with gesture control may be made with reference to theimplementation of the method as described above, and thus no morerepetition will be made here. Terms such as “unit” or “module” usedbelow may be a combination of software and/or hardware with a specifiedfunction. Although the device as described by the embodiment below ispreferably implemented with software, it is also possible to assume theimplementation with hardware or the combination of software andhardware.

As shown in FIG. 8, an device for enabling virtual reality interactionwith gesture control comprises a second image displaying module 1, apointer recognizing module 2 and an interaction displaying module 3;

the second image displaying module 1 is used for displaying aphotographed second image in response to a gesture detecting signal;

the pointer recognizing module 2 is used for recognizing an actualgesture action in the second image and converting the same into agraphic pointer; and

the interaction displaying module 3 is used for displaying a first imageand displaying the graphic pointer at a position corresponding to thefirst image for interaction.

As shown in FIG. 8, the pointer recognizing module 2 comprises acapturing and analyzing module 21, a hand shape detecting module 22 anda graphic pointer converting module 23;

the capturing and analyzing module 21 is used for capturing moving blockmasses appearing in the second image, analyzing the moving block massesto obtain several block masses approximate to a hand and selecting allthe block masses approximate to the hand as candidate hand regions;

the hand shape detecting module 22 is used for determining a hand shapein each candidate hand region, and if the plurality of candidate handregions pass verification, selecting a sum of the candidate hand regionspassing the verification as a particular region; and

the graphic pointer converting module 23 is used for converting theactual gesture action in the particular region into the graphic pointer.

Further, the interaction displaying module 3 is configured so that thephotographed second image displayed contains the first image.

Further, the interaction displaying module 3 is configured so that ‘thephotographed second image displayed contains the first image’ refersthat the first image and the second image are superimposed with an AlphaBlending algorithm.

Further, the interaction displaying module 3 is configured so that thesecond image is preprocessed as the second image is photographed.

Further, the interaction displaying module 3 is configured so that thepreprocessing performed on the second image comprises at least one ofzooming, white balance adjusting, exposure adjusting and imagede-noising.

It should be noted that the capturing and analyzing module 21 comprisesa hand region capturing module and an analyzing module; the hand regioncapturing module is used for taking the second image and capturing ahand region in the second image, more specifically, the hand regioncapturing module is a camera for the VR device; the camera is eitherplaced inside the VR device or installed outside the VR device, and thecamera includes a color camera, a depth camera, an rgb-d camera, aninfrared camera and the like, and since how to process the photographedsecond image in case of the depth camera is described in the method, nomore repetition is made here.

It should be noted that although several modules of the device forenabling virtual reality interaction with gesture control are mentionedin the detailed description above, this division is only illustrativebut not compulsory. In fact, the features and functions of three andmore modules as described above can be embodied in one module accordingto the embodiment of the present invention. On the contrary, thefeatures and functions of one module as described above can be furtherdivided into a plurality of modules for embodification.

In addition, although the operation of the method of the presentinvention is described in a particular sequence in the drawings, it isnot intended to require or imply that these operations must be executedaccording to the particular sequence, or an expected result can beachieved only after all the operations as shown are executed.Additionally or alternatively, some steps therein can be omitted, aplurality of steps can be combined into one step to be executed, and/orone step can be broken down into a plurality of steps to be executed.

The object, technical solution, and advantageous effects of the presentinvention are further illustrated in details with the specificembodiments as described above, and it should be understood that thedescription above only involves the specific embodiments of the presentinvention and is not intended to limit the protection scope of thepresent invention. It is particularly indicated that for those skilledin the art, any modifications, equivalent substitutions, improvementsand the like made within the spirit and principle of the presentinvention are construed to be incorporated within the protection scopeof the present invention.

What is claimed is:
 1. A method for enabling virtual reality interactionwith gesture control, comprising the following steps: displaying aphotographed second image in response to a gesture detecting signal;recognizing an actual gesture action in the second image and convertingthe actual gesture action into a graphic pointer; and displaying a firstimage and displaying the graphic pointer at a position corresponding tothe first image for interactions; wherein recognizing the actual gestureaction in the second image and converting the actual gesture action intothe graphic pointer comprises the following steps capturing moving blockmasses appearing in the second image, analyzing the moving block massesto obtain a plurality of block masses approximate to a hand, andselecting all the block masses approximate to the hand as candidate handregions; determining a hand shape in each candidate hand region, and ifa plurality of the candidate hand regions pass verification, selecting asum of the candidate hand regions passing the verification as aparticular region; and converting the actual gesture action in theparticular region into a graphic pointer.
 2. The method for enablingvirtual reality interaction with gesture control according to claim 1,wherein the first image is presented before the gesture detecting signalis detected.
 3. The method for enabling virtual reality interaction withgesture control according to claim 2, wherein the displayed photographedsecond image contains the first image.
 4. The method for enablingvirtual reality interaction with gesture control according to claim 3,wherein that the displayed photographed second image contains the firstimage refers that the first image and the second image are superimposedwith an Alpha Blending algorithm.
 5. The method for enabling virtualreality interaction with gesture control according to claim 4, whereinthe second image is preprocessed as the second image is photographed. 6.The method for enabling virtual reality interaction with gesture controlaccording to claim 5, wherein the preprocessing performed on the secondimage comprises at least one of zooming, white balance adjusting,exposure adjusting, and image de-noising.
 7. A device for enablingvirtual reality interaction with gesture control, comprising a secondimage displaying module, a pointer recognizing module, and aninteraction displaying module; wherein the second image displayingmodule is used for displaying a photographed second image in response toa gesture detecting signal; the pointer recognizing module is used forrecognizing an actual gesture action in the second image and convertingthe actual gesture action into a graphic pointer; and the interactiondisplaying module is used for displaying a first image and displayingthe graphic pointer at a position corresponding to the first image forinteraction; wherein recognizing the actual gesture action in the secondimage and converting the actual gesture action into the graphic pointercomprises the following steps: capturing moving block masses appearingin the second image, analyzing the moving block masses to obtain aplurality of block masses approximate to a hand, and selecting all theblock masses approximate to the hand as candidate hand regions;determining a hand shape in each candidate hand region, and if aplurality of the candidate hand regions pass verification, selecting asum of the candidate hand regions passing the verification as aparticular region; and converting the actual gesture action in theparticular region into a graphic pointer.
 8. The device for enablingvirtual reality interaction with gesture control according to claim 7,wherein the interaction displaying module is configured so that thefirst image is presented before the gesture detecting signal isdetected.
 9. The device for enabling virtual reality interaction withgesture control according to claim 8, wherein the interaction displayingmodule is configured so that the displayed photographed second imagecontains the first image.
 10. The device for enabling virtual realityinteraction with gesture control according to claim 9, wherein theinteraction displaying module is configured so that the displayedphotographed second image contains the first image refers that the firstimage and the second image are superimposed with an Alpha Blendingalgorithm.
 11. The device for enabling virtual reality interaction withgesture control according to claim 10, wherein the interactiondisplaying module is configured so that the second image is preprocessedas the second image is photographed.
 12. The device for enabling virtualreality interaction with gesture control according to claim 11, whereinthe interaction displaying module is configured so that thepreprocessing performed on the second image comprises at least one ofzooming, white balance adjusting, exposure adjusting, and imagede-noising.